What is Message Queueing?
- Message Queuing enables asynchronous communication between different components or systems, in order to increase system reliability, scalability, and flexibility.
- This is achieved by adding a message broker (like RabbitMQ or LavinMQ) to pass messages or data between processes or systems.
- Messages are sent by a producer to the Message Queue, where they are held until consumed by a consumer.
- This decouples the producer from the consumer, allowing them to operate independently.
- Message Queuing can be used for distributed computing, task scheduling, and workflow management.
Basic concepts of Message Queueing
This blog article explains Message Queuing: what it is, how to use it, and the benefits of using a Message Queue in an architecture.
A Queue is a line of things waiting to be handled, starting at the beginning of the line and processing it in sequential order. A Message Queue is a queue of messages sent between applications. It includes a sequence of work objects that are waiting to be processed.
A Message is the data transported between the sender and the receiver application; it's essentially a byte array with some headers at the top. An example of a message could be something that tells one system to start processing a task, it could contain information about a finished task or just be a plain message.
The basic architecture of a Message Queue is simple; there are client applications called producers that create messages and deliver them to the message queue. Another application, called a consumer, connects to the queue and gets the messages to be processed. Messages placed onto the queue are stored until the consumer retrieves them.
Message Queues
A Message Queue provides an asynchronous communications protocol, which is a system that puts a message onto a Message Queue and does not require an immediate response to continuing processing. Email is probably the best example of asynchronous communication. When an email is sent, the sender continues to process other things without needing an immediate response from the receiver. This way of handling messages decouples the producer from the consumer so that they do not need to interact with the message queue at the same time.
Decoupling and Scalability
Decoupling describes how much one piece of a system relies on another piece of the system. Decoupling is the process of separating functions so that they are more self-contained.
A decoupled system is achieved when two or more systems are able to communicate without being connected. The systems can remain completely autonomous and unaware of other functions. Decoupling is often a sign of a computer system that is well structured because it is easier to maintain,
If one process in a decoupled system fails to process messages from the queue, other messages can still be added to the queue and be processed when the system has recovered. You can also use a message queue to delay processing - for example, a producer posts messages to a queue. At the appointed time, the consumers start and process the messages in the queue. A queued message can be stored-and-forwarded, and the message can be redelivered until it is processed.
Instead of building one large application, it is beneficial to decouple different parts of your application and communicate between them asynchronously using message queues. This allows for different parts of the application to evolve independently, be written in different languages, and/or be maintained by separated development teams.
A message queue will keep the processes in your application separate and independent of each other. The first process will never need to invoke another process, post notifications to another process, or follow the process flow of the other processes. It can just put the message in the queue and then continue processing. The other processes can also handle their work independently, taking the messages from the queue when they are able to process them. This way of handling messages creates a system that is easy to maintain and scale.
Message Queuing - a simple use case
Imagine that you have a web service that receives many requests every second, where no request can get lost, and all requests need to be processed by a function that has a high throughput. In other words, the web service always has to be highly available and ready to receive a new request instead of being locked by the processing of previously received requests.
In this case, placing a queue between the web service and the processing service is ideal. The web service can put the "start processing" message on a queue and the other process can take and handle messages in order. The two processes are decoupled from each other and do not need to wait. If you have a lot of requests coming in a short amount of time, the processing system will be able to process them all. The queue will persist with the requests even if their number grows.
Then imagine that the business and workload are growing and the system needs to be scaled up. All that needs to be done is to add more consumers to work off the queues faster.
RabbitMQ
If you do start to consider a queue-based solution, CloudAMQP offers to host the message queue with RabbitMQ. RabbitMQ is open source message-oriented middleware that implements the Advanced Message Queuing Protocol (AMQP). AMQP has features like queuing, routing, reliability, and security. Read more about CloudAMQP here.
Questions and feedback
Hope this article helped you understand message queuing.
Please email us at support@cloudamqp.com if you have any suggestions, questions or feedback.
